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Carnegie  Mel  inn 

Software  Engineering  Institute 

Agenda 

1.  Introductions  and  Logistics 

2.  Why? 

Why  should  an  organization  attempt  to  calculate  the  ROI  of  CMMI- 
based  efforts?  Is  ROI  the  right  way  to  show  financial  value? 

3.  When? 

When  should  these  ROI  calculations  be  performed?  When  can  you 
compare  ROI  results  across  organizations? 

4.  What? 

What  information  is  needed  in  order  to  calculate  ROI?  What 
information  is  inappropriate? 

5.  How? 

How  do  organizations  calculate  ROI?  How  can  they  do  it  better? 

6.  Wrap-Up 
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1  -  Introductions  and  Logistics 
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How  often  have  you  heard  this....? 

What  is  the  value  of  process  improvement  to  our 
organization? 

Why  CMMI? 

What  should  we  focus  on? 

How  much  is  this  going  to  cost? 

What’s  in  it  for  me? 
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Overview 

This  tutorial  is  intended  to  be  a  basic  introduction  to 
calculating  the  costs  and  benefits  of  CMMI-based  process 
improvement 

We’ll  discuss  estimating,  tracking,  and  validating  ROI  over 
the  course  of  a  CMMI  improvement  effort 

You’ll  be  introduced  to  several  ways  to  calculate  ROI,  and 
the  pros  and  cons  of  each 

Implementation  approaches,  hints  and  tricks  will  be 
introduced  to  help  you  really  make  this  work! 
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Logistics 

Workshop  Time/Duration 

Rest  Rooms 

Breaks 

Smoking  Rules 

Phones 

Messages 
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ROI  can  be  calculated  many  ways 

There  is  no  single,  official  “definition”  of  ROI  -  it  can  be 
calculated  in  many  ways. 

Some  of  these  include: 

•  Benefit/Cost  ratio  (typically  considered  ROI) 

•  Net  Present  Value  (NPV) 

•  Internal  Rate  of  Return  (IRR) 

•  Payback  Period  (PP)/Break  Even  Point  (BEP) 

A  complete  ROI  analysis  includes  all  of  the  above. 
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The  Value  of  CMMI  Improvement 

Average  software  organizations  spend  65%  or  more  of  their 
engineering  dollars  addressing  quality  problems! 

This  means  that  only  1/3  of  the  organization  is  actually 
creating  something. 

Leading  software  organizations  can  reduce  this  Cost  of 
Quality  to  40%  or  less,  resulting  in  50%  or  more  gains  in 
productivity. 

Numerous  CMM  case  studies  conclusively  demonstrate  the 
value  that  can  be  achieved. 

A  rapidly  growing  number  of  CMMI  case  studies  are 
duplicating  &  exceeding  these  results. 
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Examples 

Reduced  cost  of  poor  quality  from  over  45%  to  under  30% 

•  2:1  ROI  Over  3  years  (Siemens  Information  Systems  Ltd,  India) 

$2.1  Million  in  savings  in  hardware  engineering  processes 

(Reported  under  non  disclosure) 

Reduced  software  defects  per  million  delivered  SLOC  by  over 
50%  compared  to  defects  prior  to  CM  Ml 

(Lockheed  Martin  Systems  Integration) 

Reduced  defect  rate  at  CMMI  ML5  approximately  one  third 
compared  to  performance  at  SW-CMM  ML5 

(Lockheed  Martin  Maritime  Systems  &  Sensors  -  Undersea  Systems) 

Avoided  $3.72M  in  costs  due  to  better  cost  performance 

•  As  the  organization  improved  from  SW-CMM  level  4  to  CMMI 
level  5  (Raytheon  North  Texas  Software  Engineering) 

2.5:1  ROI  over  1st  year,  with  benefits  amortized  over  less  than  6 
months  (reported  under  non  disclosure) 
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Thales  Research  &  Technology 


CMM  data  from  another  Thales  Unit  used  by  Thales 
Research  &  Technology  as  part  of  rationale  to  begin 


PI  with  CMMI. 


Results  measured  by  a  THALES  Unit  © 


Average  Cost  Variance  ■ 
reduced  by  20% 

On  average  late  to 
acceptance  -  divided  by 
24 

Number  of  defects 
during  Customer 
acceptance  -  divided  by 
20 

Cost  of  customer 
acceptance  -  reduced  by 
60% 


for  Software  impact  on  cost  deviation 

%  Project  Distribution 


96  Level  3 


Between  CMM  level  1  and  CMM  level  3 
10  software  projects  for  each  plot 


92  Level  1 


%  Cost  deviation 


Thales  Research  &  Technology  UK 


HALI 


Improvements  in: 


Product  cost 


Schedule  / 
cycle  time 


Quality 


Customer 

satisfaction 


Getting  Started  with  Process  Improvement  Using  the  CMMI®-  Carol  Marsh,  Patrick  Vigier. 
ESEPG  2003. 
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Northrop  Grumman  IT 

Appraised  at  CMMI  ML  5  in  December  2002 


Results 

•  met  25+  milestones  in  a  row 

•  earned  a  rating  of  “Exceptional”  in  every 
applicable  category  on  a  formal  Contractor 
Performance  Evaluation  Survey 


Improvements  in: 


Schedule  / 
cycle  time 


Customer 

satisfaction 


Hours  Invested:  124  in  Defect  Prevention  (CAR) 
Hours  saved:  1650  hours  (15  hours  per  defect) 

ROI:  13:1 


Quality 


Integrating  PSPsm  and  CMMI®  Level  5.  Gabriel  Hoffman,  Northrop  Grumman  IT  .  May  1, 2003 
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Accenture 

Transition  SW-CMM  to  CMMI  ML  3 

•  May  2001  to  May  2002 

•  Transition  Time:  1149  person  hours 

Key  Content 

Measurement  and  Analysis 
DAR  -►  TS,  RM,  Change  Control 
IPPD  "►  visions,  OEI 
Generic  Goals 

Results 

•  ROI:  5:1  (for  quality  activities) 


Costs: 


Improvements  in: 


Cost  of  quality 
/  ROI 


Innovation  Delivered.  CMMI®  Level  3  in  a  Large  Multi-Disciplinary  Services  Organization. 
Bengzon,  SEPG  2003 
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Some  Caveats! 

No  financial  or  accounting  knowledge  is  assumed 

The  tutorial  is  not  intended  to  be  comprehensive,  some 
topics  are  presented  at  a  high-level  only 

In  some  cases,  nuances  of  financial  accounting  and  ROI 
determination  are  not  addressed 

We’ll  do  our  best  to  get  you  started  without  burying  you  in 
details! 
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Audience 

Executive/leaders  of  organizations  seeking  to  understand  the 
costs/benefits  of  CMMI-based  process  improvement  and  how  to 
quantify  them 

Process  improvement/EPG  personnel  seeking  ways  to 
communicate  more  effectively  to  senior  management  about  the 
costs/benefits  of  CMMI-based  process  improvement 

Personnel  in  organizations  starting  down  a  CMMI-based  process 
improvement  path 

Project  managers  and  engineers  who  need  information  to  make 
better  decisions  about  their  project  work. 

Personnel  in  higher-maturity  organizations  seeking  to  transition  to 
the  CMMI,  or  implement  higher-maturity  process  areas 
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Introductions 

Tutorial  leader  introductions 

Participant  introductions 

•  Name 

•  Company 

•  Position 

•  Expectations 

-  What  do  you  want  to  get  out  of  the  tutorial? 

-  Do  your  expectations  match  the  tutorial  agenda? 

-  How  can  we  best  address  your  concerns? 
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Tutorial  Approach 

Interactive  Presentation 

Examples 

Exercises 

Questions  and  Answers 

Participate!!! 


©  2004  by  Carnegie  Mellon  University 


Version  1.1 


CMMI  ROI  Tutorial -slide  16 


Carnegie  Mellon 

Software  Engineering  Institute 

Pittsburgh,  PA  15213-3890 


2  -  Why? 

i.  To  develop  the  business  case  for  process 
improvement 

ii.  Need  to  communicate  to  decision  makers  who 
understand  the  language  of  money,  not  CMMI 

iii.  ROI  is  the  only  way  to  show  financial  value 
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Business  Context 

The  first  question  to  answer  is  “what  is  the  business  context 
in  which  we’re  undertaking  a  CMMI  effort?”  Is  it  because 
we  want  to: 

•  Do  things  better,  faster,  cheaper? 

•  Increase  contract  award  fees? 

•  Secure  more  contracts? 

•  Improve  product  quality? 

We’re  going  to  focus  on  the  quality,  project 
management,  and  productivity  implications  of  CMMI 
process  improvement 
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System  Development  Costs 

Costs  incurred  to  develop  the  product 
Examples: 

•  Requirements  development 

•  Design 

•  Product  development  +  unit 
test 

•  Project  management 

•  Engineering  infrastructure 

•  Purchased  components 

What  isn’t  on  this  list? 
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Cost  of  Quality  (CoQ)  Concept 

Developed  by  J.M.  Juran  and  applied  successfully  by 
companies  like  Toyota  Motor  Corporation  to  achieve 
competitive  advantages  through  the  development  of  better 
quality  products 


CoQ  represents  all  costs  associated  with  poor  quality 

<  Prevention  Costs 

^  Appraisal  Costs 


\  *  Internal  Failure  Costs 

Costs  Due  to  Poor  Quality 


External  Failure  Costs 


Source:  Juran’s  Quality  Handbook 
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Cost  of  Quality  Appraisal 


Objectives 

•  Quantify  quality  costs  in  the 
language  management 
understands  -  money 

•  Identify  major  opportunities 
for  quality  cost  reduction 

•  Identify  opportunities  for 
reducing  customer 
dissatisfaction  and  threats  to 
product  saleability 

•  Expand  budgetary  and  cost 
controls 

•  Stimulate  improvement 
through  publication 


•  Improve  ability  to  deliver 
more  to  the  business/market 

•  Reduce  the  functionality  gap 
versus  the  competition 

•  Provide  executives  with 
reasons  to  invest  in 
engineering  improvement 

•  CoQ  appraisals  are  an  easy 
way  to  understand  where 
money  is  spent  and  to 
gauge  the  effectiveness  of 
an  organization 
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Internal  Failure  Costs 

Costs  associated  with  defects  that  are  found  prior  to  transfer 
of  the  product  to  the  customer 


Examples: 

•  Design  corrective  action 

•  Design  re-reviews 

•  Purchased  component 
corrective  action 

•  Purchased  component  re¬ 
test 

•  Defect  reporting/tracking 

•  Defect  fixing 

•  2nd  and  subsequent 
integration  testing  iterations 

•  2nd  and  subsequent  system 
testing  iterations 
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External  Failure  Costs 

Costs  associated  with  defects  that  are  found  after  the 
product  is  shipped  to  the  customer 

Examples: 


•  Next  release  defect  rework 
(maintenance) 

•  “Re-engineering” 

•  Technical  support  personnel 

•  Product  returns 

•  Lawsuits 

•  Contract  penalties 

•  Lost  customers 


•  Lower  marketplace 
perception 

•  Loss  of  pricing  power 

•  Lost  sales 
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Graphical  Representation  of  CoQ 
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Prevention  Costs 

Costs  incurred  to  keep  failure  and  (product)  appraisal  costs 
to  a  minimum 


Examples: 

•  Quality  planning 

•  Quality  assurance 

•  Configuration  management 

•  Supplier  capability 
assessments 

•  Quality  training 

•  Component/software  reuse 


•  Requirements  reviews 

•  Design  reviews 

•  Code  reviews 

•  CM  tools 

•  External  process 
appraisals/audits  (CMMI, 
ISO,  SPICE,  etc) 

•  Process  improvement  efforts 
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(Product)  Appraisal  Costs 

Costs  incurred  to  determine  the  degree  of  conformance  to 
quality  requirements 

Examples: 


Purchased  component 
testing 

Defect  reporting/tracking 
Test  automation 
systems/software 


First  iteration  integration 
testing 

First  iteration  system  testing 
User  acceptance  testing 
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Relating  CoQ  to  Productivity 

Decreasing  an  engineering  organization’s  Cost  of  Quality 

directly  impacts  productivity: 

•  Fewer  defects  mean  less  time  testing  and  reworking  the 
product 

•  Less  time  testing  means  a  shorter  cycle  time 

•  Shorter  cycle  time  means  more  products  can  be 
developed  in  similar  time  frames 

•  Fewer  defects  also  mean  a  smaller  customer  support 
staff  -  these  resources  can  be  applied  to  new  product 
development 
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SW  Example:  65%  More  Productivity 


Average 


Leading 


Requirements 

4.5  Months 

Design 

6.6  Months 

Requirements 
4.0  Months 

Design  | 

6.0  Months  | 

Development 
8.8  Months 


Test 

9.4  Months 


Development 
6.8  Months 


Test 

3.75  Months 


Leading  organizations 
spend  20%  less  on  software 
development  effort  and 
schedule,  and  50%  less  on 
testing  costs! 


20% 


Rework 


Leading  organizations 
deliver  80%  less  defects  to 
production,  resulting  in 
significantly  less  rework 
and  45%  more  productivity! 


45% 


Average  organizations  spend  30%  of  project  resources  on  testing,  leading  organizations  spend  15% 
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Leverage 


Process  improvement  has  a  leverage 
effect  beyond  just  quality  and  productivity 
increases 


Current 

Capability 


Improved 

Capability 


Higher  Revenue  and 
Profits  through 
More  Marketable 
Products 
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Communication 

Executives,  senior  management,  and  other  decision 
makers  don’t  use  CMMI  language 

$  Their  language  is  money 

$  Decision  making  is  done  in  the  context  of  money 

so . 

$  Communicating  the  importance  and  effectiveness  of 
CMM-based  process  improvement 

$  Champions  must  communicate  in  the  language  of 
executives  -  money 
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Business  Case 

•  Resources  are  scarce  (especially  $money$) 

•  It  is  impossible  to  execute  on  all  opportunities  available 
to  the  organization 

•  Executives  want  an  “apples-to-apples”  comparison  of  the 
various  investment  options  available 

Incorporating  ROI  analysis  into  a  proposed  or 
existing  CMMI  effort  allows  for  comparison  of 
efforts  across  seemingly  disparate  projects 

And,  the  ROI  of  CMMI  can  be  quite  high 
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Provide  Focus  and  Tracking  Capability 

Performing  an  ROI  analysis  up-front  (estimation)  can 

•  focus  the  organization  on  concrete  business  objectives 
for  CMMI  process  improvement, 

•  clarify  how  improvement  efforts  map  to  corporate 
objectives, 

•  establish  expectations  about  results 

•  establish  a  baseline  to  track  performance 


Multi-year  process  improvement  efforts  need  to  maintain 
the  business  focus  by  continuously  tracking 

$  the  associated  actual  costs 

$  the  related  benefits 
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Drive  Better  Results 
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Difficulties  in  Using  ROI 

Differing  perceptions  of  ROI 
Buy-In 

Access  to  Data 
Calculating  benefits  is  tough 
Consistently  measuring  ROI  over  time  is  tough 
Organizations  are  resistant  to  measurement 
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Discussion 


What  other 
difficulties  have 
you 

experienced  in 
determining 
CMMI  ROI? 
How  did  you 
overcome 
them? 
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A  Process  for  Determining  ROI 


Determine 
Effort  Scope/ 
Current  Needs 

C  Estimate/ 
Determine 
Effort  Costs 

f  Estimate/ 

(  Determine 

V  -  Effort  Benefits 

\ 

Estimate/ 
Calculate  ROI 

\ 

Present 
Results  to 
Management 

I 


Approve  and  execute  effort 


Continued  process 
improvement 


Track  actual 
costs/benefits  to 
determine  actual 
ROI 
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3  -  When? 

i.  Before 

ii.  During 

iii.  After 
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Before 

Initially,  it  is  VERY  important  to  understand  the  potential 
value  of  a  CMMI-based  improvement  effort  to: 

•  Clearly  understand  the  estimated  costs  associated  with 
the  effort 

•  Demonstrate  the  potential  business  value 

•  Provide  a  comparison  against  competing  projects 

•  Establish  a  baseline  for  ongoing  tracking  and  oversight 

•  Establish  the  expectations  that  benefits  require  time  to 
develop,  particularly  for  senior  management 

•  Communicate  with  decision  makers 

At  this  time,  other  organizational  indicators  such  as  quality 
measurements,  schedule  adherence,  customer  satisfaction, 
etc.  should  be  baselined. 
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During 

In  order  to  track  and  maintain  visibility,  ROI  should  be 
tracked  on  an  ongoing  basis  (perhaps  every  6-12  months) 

At  the  same  time,  intermediate  indicators  such  as  quality 
measurements,  customer  satisfactions  surveys,  etc.  are 
tracked  to  determine  the  impact  of  improvement  activities 

Doing  so  demonstrates  that  the  effort  is  on  track  from  a 
cost  perspective,  and  continues  to  keep  the  focus  on  the 
effort. 

Additionally,  early  wins  are  communicated 
unambiguously  in  financial  terms  -  something 
executives  like  to  see. 
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After 

After  a  particular  effort  is  completed,  the  ROI  should  be 
calculated  to  fully  understand  the  true  value  of  the  effort. 

This  is  needed  to  evaluate  the  financial  effectiveness  of  the 
effort,  create  lessons  learned,  and  set  the  stage  for  driving 
continuous  improvement  and  institutionalization. 

This  forms  the  basis  for  the  implementation  of  higher 
maturity  process  areas  that  support  fact-based  decision 
making. 
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In  other  words.... 

continuously, 

throughout  the  entire  effort! 


Implementation  of  the  Measurement  and  Analysis  process 
area  early  in  your  adoption  effort  will  facilitate  this.  It  will 
also  support  your  organization’s  comparison  to  applicable 
benchmarks  as  well  as  modeling  and  simulation. 
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4  -  What? 

i.  All  applicable  direct  costs  should  be 
identified 

ii.  As  well,  all  direct  benefits  should  be 
accounted  for 

iii.  Indirect  costs  and  benefits  shouldn’t  be 
part  of  the  analysis  as  they  are  difficult  to 
attribute  solely  to  the  effort 
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What  information  is  needed . 

In  its  most  basic  definition  ROI  =  A  comparison  of  the  costs 
and  benefits  of  a  process  improvement  effort  across  a 
specific  organizational  scope  and  time  span. 

You  need  to  understand: 

•  Scope  of  the  analysis 

•  Appropriate  time  horizon  for  analysis 

•  Relevant  costs 

•  Financially  quantifiable  benefits 

Remember,  for  a  proper  ROI  analysis,  all  cost  and 
benefit  information  needs  to  be  calculated  in  terms 
of  dollars  and  cents. 
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Process  Improvement  ROI  Model 


Costs 


Additional 

Personnel 

Project  Time 
Associated  with 
PI 

Training 

Software  Licenses 
Consulting  Help 

Assessments/ 

Appraisals 


Non-Financially 
Quantifiable 
Primary  Benefits 


Financially 
Quantifiable 
Primary  Benefits 


Improved  Quality 

Process  Capability 
and  Organizational 
Maturity 

Schedule  Adherence 


Reduced  Cost  of 
Quality  Resulting 
in  Productivity 
Increases 

•  more  features/ 
functionality 
delivered  for 
the  same  price 

•  same  features/ 
functionality 
delivered  for  a 
lesser  price 


Difficult  to  Attribute 
Secondary 
Business  Benefits 


Improved  Customer 
Satisfaction 
Improved  Employee 
Satisfaction 
Lower  Training 
Costs 

Increased  Sales/ 
Market  Share 
Improved  Award 
Fees 
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Scope  of  Analysis 

There  are  many  ways  to  scope  an  ROI  analysis. 

•  At  the: 

-  Organizational  level 

-  Project  level 

-  Process  level 

•  Examples  include: 

-  Moving  from  CMM  to  CMMI 

-  Introducing  all  practices  associated  with  a  particular  PA 

-  Implementing  specific  practices 

-  Achieving  a  specific  maturity  or  capability  level 

•  ROI  calculations/equations  won’t  change  due  to 
scope. 
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Time  Horizon 

Over  how  much  time  should  an  ROI  analysis  be 
performed? 

Organizational  -  Generally,  organizational  efforts  must 
show  net  value  over  a  2-3  year  time  period  and 
unambiguous  incremental  value  in  shorter  time  frames. 

Project/process  -  must  show  value  in  a  shorter  period  of 
time. 

It  is  probably  unrealistic  in  most  circumstances  to 
demonstrate  meaningful  and  measurable  benefit  in  less 
than  3  months  time  due  to: 

•  Need  to  develop/implement  processes 

•  Train  organizational  units 

•  Institutionalize  the  improved  processes 
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Scope:  Exercise 

At  this  point,  we’ll  begin  working  though  our  fictitious 
example  of: 

•  Organizational  CMMI  improvement  effort 

•  150  person  engineering  organization  at  maturity  level  1 

•  Looking  to  reach  maturity  level  2  in  18  months 

•  No  EPG  or  QA  in  place 

•  Experiencing  a  Cost  of  Quality  of  65% 

In  the  exercise,  we’ll  utilize  an  ROI  calculation  worksheet 
that  guides  the  type  of  information  we  need  to  consider. 

First  off,  we  need  to  define  the  scope  of  the  effort 
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What  Costs  and  Benefits? 

What  costs  do  you  need  to  track? 

What  benefits  are  you  expecting  to  realize? 

When  will  costs  and  benefits  be  incurred? 
How  will  these  come  together  to  generate  ROI? 
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Key  Factors 

Generating  Cost  &  Benefit  Data,  you  need  to  be  aware  of: 

•  Consistency 

-  in  gathering  over  time 

-  in  matching  costs  and  benefits 

•  Clear  &  detailed  operational  definitions 

•  Clearly  understanding  incremental  costs  and  benefits 

•  Costs  and  benefits  defined  in  financial  terms 
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Direct  People  Costs  of  CMMI  Improvement 


Costs  of  Process  Definition  /  Improvement 

Process  definition  & 
documentation 

Interviews,  meetings 

MSG,  EPG,  TWGs 

Attendance  in  training 

Management  of  the  effort 

Process  implementation 

Consulting 

Process  institutionalization 

Additional  personnel 

Process  adherence 

Project  team  diversion 

Assessments/appraisals 

Communication 

Rewards  and  recognition 

Generally  counted  as  salary  &  benefit  costs  associated 
with  people’s  time 
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Direct  Non-People  Costs  of  CMMI 
Improvement 


Costs  of  Process  Definition  /  Improvement 

Tools 

PAL,  Measurement 

Repository 

Training 
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Indirect  Costs  of  CMMI 
Improvement 


Costs  of  Process  Definition  /  Improvement 

Loss  of  organizational/project 
team  productivity 
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Other  Costs  to  Think  About 


Rework  due  to  poor  quality 

•  Prior  to  shipping 

•  After  shipping 

Poor  project  cycle  time 
Lack  of  schedule  predictability 


Rather  than  consider  these  expenses  on  the  “cost” 
side  of  the  ROI  analysis,  we’ll  consider  improving  them 
to  be  part  of  the  “benefits”  side. 
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Costs:  Exercise 

•  Organizational  CMMI  improvement  effort 

•  150  person  engineering  organization 

•  Looking  to  reach  maturity  level  2  in  18  months 

•  No  EPG  or  QA  in  place 

•  Experiencing  a  Cost  of  Quality  of  65% 

Which  costs  should  we  plan  and  account  for? 
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Direct  Benefits 


Reductions  in: 

Improvements  in: 

Defects 

Process  adherence 

Rework 

Quality 

Effort/overtime 

Product  features 

Schedule  cycle  time 

Product  costs 

Production  costs 

Support  costs 
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Indirect  Benefits 


Reductions  in: 

Improvements  in: 

Training  costs 

Employee 

satisfaction/retention 

Customer 

satisfaction/retention 

Revenue 

Award  fees 

Market  share 
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Discussion 


Most  direct 
process 
improvement 
benefits  can  be 
financially 
quantified  in 
terms  of 
productivity 
increases 
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So,  How  Do  We  Measure  Productivity? 

The  definition  of  Productivity  is  the  number  of  outputs 
delivered  for  a  certain  number  of  inputs. 

For  systems  engineering,  the  inputs  are  typically  people- 
related. 

The  outputs  however,  can  vary  significantly,  and  are 
difficult  to  compare  against  each  other: 

•  Missile  system 

•  Copy  machine 

•  Cell  phone 

•  Function  points  or  SLOC  (for  software) 

The  easiest  way  to  help  understand  the  notion  of 
productivity  in  an  engineering  organization  is  Juran’s 
Cost  of  Quality  concept. 
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Measuring  Non-Financial  Benefits 

As  discussed,  there  are  many  non-financial  benefits  with 
different  ways  to  measure  them. 

It  is  important  to  measure  non-financial  benefits  on  an 
ongoing  basis  as  part  of  the  effort  in  order  to  keep 
focus  on  other  indicators  of  progress  or  value. 

Keep  in  mind  that  measuring  these  in  non-financial  terms 
doesn’t  directly  impact  the  ROI  calculation. 
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Using  Benchmarks  to  Estimate 

To  aid  in  determining  what  costs  and  benefits  may  be 
expected,  consider  the  use  of  industry  benchmarks.  For 
example: 

•  Gartner  research  shows  software  organizations  spend 
3%  -  5%  on  CMMI  process  improvement  efforts 

•  CMM  maturity  level  1  organizations  typically  experience 
a  Cost  of  Quality  of  65% 

•  CMM  maturity  level  3  organizations  typically  experience 
a  Cost  of  Quality  of  40% 

Baselining  the  organization  and  using  benchmarks  to 
compare  against  helps  in  building  a  credible  ROI 
calculation. 
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Using  Simulation  to  Estimate 

Engineering  process  simulation  models  focus  on  the 
dynamics  of  engineering  development  and  maintenance. 

Modeling  and  simulation  of  the  significant  features  of 
engineering  processes  facilitates  experimentation,  answer 
“what-if  questions,  and  perform  sensitivity  analysis. 

By  modeling  an  organization’s  current  engineering 
processes  and  simulating  improvements  to  them,  costs  and 
benefits  can  be  better  estimated  and  determined. 

Example:  would  the  cost  of  source  code  inspections  add 
value  if  the  input  source  code  had  an  abnormally  high 
degree  of  errors? 
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Benefits:  Exercise 

•  Organizational  CMMI  improvement  effort 

•  150  person  engineering  organization 

•  Looking  to  reach  maturity  level  2  in  18  months 

•  No  EPG  or  QA  in  place 

•  Experiencing  a  Cost  of  Quality  of  65% 

Benchmarks  indicate  that  an  organization  operating  at 
maturity  level  2  can  reduce  its  Cost  of  Quality  to  50% 

How  can  we  quantify  these  benefits? 
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Levels  of  Data  Fidelity 

The  level  of  detail  you  go  to  in  an  ROI  analysis  will  depend 
on  the  size  of  the  effort 

Multi-million  dollar  CMMI  efforts  in  larger  organizations  will 
most  likely  require  a  greater  degree  of  data  fidelity  than  a 
smaller,  more  tactical  solution  on  just  one  project. 

Understand  the  level  of  fidelity  needed  to  make  a  credible 
ROI  analysis.  Don’t  blow  your  chance  with  a  shallow  case 
if  decision  makers  are  expecting  more  details. 

“Test  run”  your  assumptions  and  analysis  with  peers 
and  other  managers  before  presenting  the  final  results. 
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5  -  How? 


i.  There  are  several  ways  to  calculate  ROI 

ii.  Using  multiple  methods  provides  for  a  multi¬ 
faceted  view  and  allows  for  better  decision 
making 
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ROI  can  be  calculated  many  ways 

There  is  no  single,  official  “definition”  of  ROI  -  it  can  be 
calculated  in  many  ways. 

Some  of  these  include: 

•  Benefit/Cost  ratio  (typically  considered  ROI) 

•  Net  Present  Value  (NPV) 

•  Internal  Rate  of  Return  (IRR) 

•  Payback  Period  (PP)/Break  Even  Point  (BEP) 

A  complete  ROI  analysis  includes  all  of  the  above 
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Match  Cash  Flows  and  Don’t  Double 
Count 

Costs  must  align  appropriately  with  benefits 

•  Don’t  include  costs  or  benefits  that  don’t  relate  to  each 
other 

•  Beware  of  “one-time”  costs  that  should  really  be  incurred 
over  the  entire  ROI  analysis  time  period 

•  Don’t  scope  the  effort’s  costs  to  a  specific  project  but 
count  benefits  organizationally 

No  double  counting 

•  For  example,  you  can’t  count  the  value  of  an  avoided 
defect  and  the  productivity  that’s  improved  as  a  result 
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Benefit/Cost  Ratio 

Benefit/Cost  Ratio  (typically  considered  to  be  ROI)  is  simply 
the  ratio  of  benefits  enjoyed  by  a  certain  effort  divided  by 
the  costs  incurred,  i.e.  150% 

Pros 

•  Fairly  easy  to  compute 

•  Provides  a  “one  number”  method  of  comparing  projects 
Cons 

•  Doesn’t  incorporate  the  Time  Value  of  Money  (TVM) 

•  No  way  to  know  the  dollar  magnitude  of  the  return 

•  Don’t  know  how  long  the  return  took  to  achieve 
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Benefit/Cost  Ratio 


_ Year  0 _ Year  1 _ Year  2 _ Total 

Costs  $300  $100  $100  $500 

Benefits  $0 _ $500 _ $500 _ $1000 

Return  -$300  $400  $400  $500 

Benefit/Cost  Ratio  in  this  example  would  be  2:1  or  200% 
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Time  Value  of  Money  (TVM) 

Benefit/Cost  ratio,  while  valuable,  doesn’t  take  into  account 
the  concept  of  the  Time  Value  of  Money  (TVM). 

Simply  stated,  the  value  of  a  dollar  is  less  in  the  future  than 
it  is  now. 

Question:  If  risk-free  interest  rates  are  10%,  how  much 
would  you  pay  me  now  for  $100  in  3  years? 
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Time  Value  of  Money  (TVM) 


Answer:  $75.13 

_ Year  0 _ Year  1 _ Year  2 

Principal  $75.13  $82.64  $90.91 

Interest  $7.51 _ $8.26 _ $9.09 

Total  $82.64  $90.91  $100 
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Discounted  Cash  Flow  (DCF) 

To  aid  in  corporate  decision-making,  the  TVM  is  captured  in 
Discounted  Cash  Flow  (DCF)  analysis. 

In  DCF  analysis,  future  cash  flows  are  discounted  using 
either: 

•  The  company’s  weighted  average  cost  of  capital 
(WACC) 

•  A  “hurdle  rate”  consisting  of  the  company’s  cost  of 
capital  adjusted  by  a  risk  premium  appropriate  for  the 
type  of  project  being  analyzed 
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Discounted  Cash  Flow  (DCF) 

This  risk  premium  can  be  significant  (30%+)  in  some  cases! 

_ Year  0 _ Year  1 _ Year  2 

Principal  $45.52  $59.17  $76.92 

Interest  $13.65 _ $17.75 _ $23.08 

Total  $59.17  $76.92  $100 
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Net  Present  Value  (NPV) 

Net  Present  Value  (NPV)  takes  both  cost  and  benefit  cash 
flows  and  discounts  them  appropriately  using  the 
company’s  cost  of  capital  or  hurdle  rate. 

NPV  is  described  as  a  dollar  value,  allowing  for  seemingly 
dissimilar  projects  to  be  compared  against  each  other. 

Pros 

•  Articulates  the  dollar  magnitude  of  the  return 

•  Provides  a  “one  number”  method  of  comparing  projects 

•  Incorporates  TVM 

Cons 

•  Not  as  easy  to  compute 

•  Don’t  know  how  long  the  return  took  to  achieve 
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Net  Present  Value  (NPV) 

With  a  10%  hurdle  rate,  our  example  looks  like: 

_ Year  0 _ Year  1 _ Year  2 _ Total 

Costs  $300  $100  $100  $500 

Benefits  $0 _ $500 _ $500 _ $1000 

Return  -$300  $400  $400  $500 

Disc  Return  -$300  $363.63  $330.58  $393.94 

(NPV) 
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Internal  Rate  of  Return  (IRR) 

IRR  determines  the  intrinsic  interest  rate  that  summarizes  the 
value  of  a  series  of  cash  flows.  0%  indicates  “break-even”, 
positive  percentage  indicates  a  positive  return  while  a  negative 
percentage  indicates  a  negative  return. 

Essentially  determines  the  discount  rate  that  results  in  NPV=0 

Pros 

•  Provides  a  “one  number”  method  of  comparing  projects  that  is 
independent  of  the  company’s  WACC  or  hurdle  rate 

Cons 

•  No  way  to  know  the  dollar  magnitude  of  the  return 

•  Don’t  know  how  long  the  return  took  to  achieve 

•  In  unusual  cases,  may  provide  multiple  answers 

•  Not  as  easy  to  compute 
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Internal  Rate  of  Return  (IRR) 

In  our  example,  the  IRR  is  100% 

_ Year  0 _ Year  1 _ Year  2 _ Total 

Costs  $300  $100  $100  $500 

Benefits  $0 _ $500 _ $500 _ $1000 

Return  -$300  $400  $400  $500 

Disc  Return -$300  $200  $100  $0 
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Payback  Period/Break  Even  Point 

Determines  the  amount  of  time  it  takes  for  cash  inflows  to 
equal  cash  outflows,  resulting  in  the  break  even  point 

Pros 

•  Indicates  how  quickly  a  project  turns  profitable 
Cons 

•  No  way  to  know  the  dollar  magnitude  of  the  return 

•  In  unusual  cases,  may  provide  multiple  answers 


©  2004  by  Carnegie  Mellon  University 


Version  1.1 


CMMI  ROI  Tutorial  -  slide  77 


Carnegie  Mel  inn 

Software  Engineering  Institute 

Payback  Period/Break  Even  Point 

In  our  example,  the  PP/BEP  is  Year  1 


_ Year  0 _ Year  1 _ Year  2 _ Total 

Costs  $300  $100  $100  $500 

Benefits  $0 _ $500 _ $500 _ $1000 

Return  -$300  $400  $400  $500 

Net  Return  -$300  $100  $500  $500 
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Exercise 

•  Organizational  CMMI  improvement  effort 

•  150  person  engineering  organization  at  maturity  level  1 

•  Looking  to  reach  maturity  level  2  in  18  months 

•  No  EPG  or  QA  in  place 

•  Experiencing  a  Cost  of  Quality  of  65% 

How  do  the  various  ROI  calculations  turn  out  for  our 
exercise? 
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6  -  Wrap-up 

Sponsored  by  the  U.S.  Department  of  Defense 
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Implementation  Issues 

ROI  analysis  won’t  necessarily  improve  a  CMMI  effort’s 
results  -  you  still  have  to  execute! 

In  any  process  improvement  effort,  you  should  find  ways  to 
leverage  organizational  change  best  practices  to  overcome 
resistance  to  change. 

Determining  and  tracking  ROI  as  well  as  other 
measurements  keeps  the  focus  of  the  effort  on  the 

business  benefits 

Use  the  Measurement  and  Analysis  process  area  early  to 
put  the  focus  on  the  business  benefits  from  the  start 
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Discussion 
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What  Did  You  Learn? 

1.  Why? 

i.  To  develop  the  business  case  for  process  improvement 

ii.  Need  to  communicate  to  decision  makers  who  understand  the  language  of  money,  not 
CMMI 

iii.  ROI  is  the  only  way  to  show  financial  value 

2.  When? 

i.  Before,  During,  and  After 

3.  What? 

i.  All  applicable  direct  costs  should  be  identified 

ii.  As  well,  all  direct  benefits  should  be  accounted  for 

iii.  Indirect  costs  and  benefits  shouldn’t  be  part  of  the  analysis  as  they  are  difficult  to 
attribute  solely  to  the  effort 

4.  How? 

i.  There  are  several  ways  to  calculate  ROI 

ii.  Using  multiple  methods  provides  for  a  multi-faceted  view  and  allows  for  better  decision 
making 

5.  Case  Study 

We  went  through  a  case-study  where  we  estimated  CMMI-based  process  improvement  costs 
and  returns  over  a  3  year  time  period,  and  explored  the  various  ways  of  calculating  the 
value  of  the  effort  over  the  3  years. 
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A  Process  for  Determining  ROI 

Determine 
Effort  Scope/ 

Current  Needs 

C  Estimate/ 

Determine 
Effort  Costs 

f  Estimate/ 

(  Determine 

V  -  Effort  Benefits 

\ 

Estimate/ 

Calculate  ROI 

\ 

Present 
Results  to 
Management 

I 

Approve  and  execute  effort 


Continued  process 
improvement 
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Parting  thoughts: 

What’s  Next  for  You?! 


Determining  the  monetary  value  of  CMMI-based  process 
improvement  may  appear  to  be  difficult. 

But  it’s  not  that  hard. 

•  It  does  take  disciplined  effort 

•  You  may  have  to  build  or  buy  the  necessary 
organizational  capabilities  &  resource  capacities 

And  it’s  extremely  important. 

•  Making  informed  decisions  about  how  to  invest  scarce 
resources 

•  Sustaining  commitment  for  disciplined  process 
improvement 
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Thank  You! 

Rolf  W.  Reitzig  -  cognence,  inc. 

-  Resident  SEI  Affiliate 

-  Rolf  Reitziq@coqnence.com 

Dennis  R.  Goldenson  -  SEI 

-  dg@sei.cmu.edu 

Diane  Gibson  -  SEI 

-  dlg@sei.cmu.edu 

Mark  R.  Cavanaugh  -  IBM  Global  Services 

-  Resident  SEI  Affiliate 

-  markrc@us.ibm.com 
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